Apparatus and method for estimating own vehicle behavior

ABSTRACT

In an apparatus for estimating a behavior of a vehicle carrying the apparatus based on images of surroundings of the vehicle captured by an imaging device, an information acquirer acquires beforehand specific location information that is information representing a specific location in which a situation around the vehicle is such that the estimation of the own vehicle behavior based on the images is unstable. In the apparatus, a behavior estimator estimates the own vehicle behavior based on the images captured by the imaging device and the specific location information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims the benefit of priority fromearlier Japanese Patent Application No. 2018-208879 filed Nov. 6, 2018,the description of which is incorporated herein by reference.

BACKGROUND Technical Field

The present disclosure relates to an apparatus and a method forestimating an own vehicle behavior.

Related Art

A system is known that records position information such as landmarksusing an image captured by a camera mounted to a vehicle, uploads theinformation to a server or the like to generate a sparse map, anddownloads the generated sparse map to determine a position of the ownvehicle when the vehicle is traveling.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a diagram schematically illustrating a configuration of a mapsystem according to a first embodiment;

FIG. 2 is a diagram for explaining a process by an SFM recognizeraccording to the first embodiment, and schematically illustrates animage captured at a predetermined time t1;

FIG. 3 is a diagram for explaining the process by the SFM recognizeraccording to the first embodiment, and schematically illustrates animage captured at a predetermined time t2;

FIG. 4 is a diagram for explaining a mechanism of aliasing occurrenceaccording to the first embodiment, and schematically illustrates animage captured at a predetermined time t11;

FIG. 5 is a diagram for explaining a mechanism of aliasing occurrenceaccording to the first embodiment, and schematically illustrates animage captured at a predetermined time t12;

FIG. 6 is a diagram for explaining a process for suppressing occurrenceof erroneous estimation due to aliasing according to a secondembodiment, and schematically illustrates an image captured at apredetermined time t11; and

FIG. 7 is a diagram schematically illustrating a plan view of the imageof FIG. 6 as viewed from above a vehicle.

DESCRIPTION OF SPECIFIC EMBODIMENT

In the system set forth above, as described in Japanese Translation ofPCT International Application Publication No. JP-T-2018-510373, whengenerating data to be uploaded to the server, that is, probe data on thevehicle side, a Structure From Motion method is used. Hereinafter,Structure From Motion is abbreviated as SFM. A technique for estimatingan own vehicle behavior with high accuracy using an optical flow such asSFM is an important technique indispensable for map generation andlocalization. The optical flow is a vector representing a motion of anobject in an image between two times and is used to find a movement offeature points from continuously captured images.

However, the estimation of ego-motion in the optical flow may beunstable depending on the surrounding environment of the vehicle. Notethat the ego-motion is a parameter that represents an attitude of acamera mounted to the vehicle, and thus an attitude of the vehicle perse. For example, on a highway or the like, when the vehicle is travelingin a section where soundproof walls of the same shape are continuouslyprovided, depending on the movement amount of the own vehicle and therepetition interval of the pattern formed by the soundproof walls, thereis a possibility that aliasing will occur, and the estimation ofego-motion in the optical flow and thus the estimation of the ownvehicle behavior will be impossible. Note that aliasing means that, forexample, an erroneous result is obtained that an object is stopped eventhough the object is actually in motion.

In view of the above, it is desired to have own vehicle behaviorestimation techniques, by which an own vehicle behavior can be estimatedaccurately even in a location where the estimation of ego-motion in animage becomes unstable.

Hereinafter, a plurality of embodiments will be described with referenceto the drawings. In each embodiment, substantially the same componentsare denoted by the same reference signs and description thereof isomitted.

First Embodiment

Hereinafter, the first embodiment will be described with reference toFIGS. 1-5.

A map system 1 illustrated in FIG. 1 is a map system for autonomousnavigation. The map system 1 additionally functions with respect to aconventional function for specifying a position of an own vehicle, suchas GPS, and is effective in identifying the position with higheraccuracy. The map system 1 is roughly divided into two functions of maputilization and map update.

In the map utilization, map information stored in a server 2 isdownloaded to a vehicle, and the vehicle identifies a position of theown vehicle based on the downloaded map information and positions oflandmarks such as signs included in an image captured by an image sensor3 such as a camera. Hereinafter, the map information stored in theserver 2 is also referred to as an integrated map. In this case, drivingsupport is achieved by, a vehicle controller 4 outputting acorresponding command to an actuator that is for operating hardwaremounted to the vehicle, based on the identified current position of theown vehicle. The actuator is a device for controlling the vehicle byhardware, such as a brake, a throttle, a steering wheel, and a lamp.

On the other hand, in the map update, information acquired by varioussensors such as the image sensor 3, a vehicle speed sensor 5, and amillimeter wave sensor (not illustrated) mounted to the vehicle isuploaded to the server 2 as probe data, and the integrated map in theserver 2 is sequentially updated. Thus, for example, driving support andautomatic steering are achieved while the position of the vehicle isalways identified with high accuracy based on the latest mapinformation.

In the map system 1, a human machine interface 6 is a user interface fornotifying a user of various types of information and for transmitting apredetermined operation to the vehicle by the user. In the followingdescription and FIG. 1 and the like, the human machine interface isabbreviated as an HMI. The HMI 6 includes, for example, a displayattached to a car navigation device, a display built in an instrumentpanel, a head-up display projected on a windshield, a microphone, aspeaker, and the like. Furthermore, a mobile terminal such as asmartphone that is communicably connected to the vehicle can also be theHMI 6 in the map system 1.

In addition to visually acquiring information displayed on the HMI 6,the user can acquire information by voice, warning sound, and vibration.In addition, the user can request a desired operation from the vehicleby a touch operation on the display or by voice. For example, when theuser intends to receive an advanced driving support service such asautomatic steering using map information, the user activates theaforementioned function via the HMI 6. For example, when a “map link”button indicated on the display is tapped, a map utilization function isactivated and download of map information is started.

In another example, the map utilization function is enabled by giving acommand by voice. In addition, upload of map information related to themap update may be performed at all times while the communication betweenthe vehicle and the server 2 is established, may be executed while themap utilization function is enabled by tapping the “map link” button, ormay be enabled by an another UI that reflects the user's intention.

The map system 1 of the present embodiment includes the server 2, theimage sensor 3, a vehicle controller 4, the vehicle speed sensor 5, theHMI 6, a GPS receiver 7, a control device 8, and the like. The server 2is provided at a location isolated from the vehicle on which the imagesensor 3 and the like are mounted. The server 2 includes a controldevice 9. The control device 9 is mainly composed of a microcomputerhaving a central processing unit (CPU), a read-only memory (ROM), arandom-access memory (RAM), an input/output (I/O) interface, and thelike, and includes an integrator 10 and an updater 11. Each of thesefunctional blocks is implemented by the CPU of the control device 9executing a computer program stored in a non-transitional tangiblestorage medium, thereby executing a process corresponding to thecomputer program. That is, each of these functional blocks isimplemented by software. The integrator 10 and the updater 11 are forexecuting various processes related to the map update described above,and details thereof will be described later.

The GPS receiver 7 outputs data Da representing GPS informationrepresented by a signal received via a GPS antenna (not illustrated) tothe control device 8 or the like. The vehicle speed sensor 5 detects avehicle speed that is a speed of the vehicle, and is configured as awheel speed sensor that detects a speed of the wheels of the vehicle.The vehicle speed sensor 5 outputs a signal Sa representing the detectedspeed, which is the detected value, to the control device 8 or the like.

The image sensor 3 is an imaging device that is mounted to a vehicle andcaptures an image of an environment around the vehicle, specifically, apredetermined range of environment in a forward travel direction of thevehicle. In addition, the image sensor 3 is not limited to the one thatcaptures an image of the forward travel direction of the vehicle, andmay be one that captures an image of the rear and the side, for example.Information on the environment around the vehicle captured by the imagesensor 3 is stored in a memory (not illustrated) in a form of a stillimage or a moving image (hereinafter collectively referred to as animage). The control device 8 is configured to be able to read data Dbstored in the memory, and executes various processes based on the dataDb.

The control device 8 is mainly composed of a microcomputer having a CPU,a ROM, a RAM, an input/output (I/O) interface, and the like. The controldevice 8 includes functional blocks such as a scale factor corrector 12,an ego-motion calculator 13, a landmark detector 14, a map generator 15,a localizer 16, and a canceller 17. Each of these functional blocks isimplemented by the CPU of the control device 8 reading and executing acomputer program stored in a storage device 22 that is anon-transitional tangible storage medium, thereby executing a processcorresponding to the computer program. That is, each of these functionalblocks is implemented by software. In addition, the storage device 22 isa memory, for example, and is mounted to the vehicle together with theimage sensor 3, the control device 8, and the like.

The control device 8 corresponds to a computing device and functions asan apparatus for estimating an own vehicle behavior that is a behaviorof a vehicle based on an image captured by the image sensor 3.Therefore, the computer program executed by the microcomputer of thecontrol device 8 includes an own vehicle behavior estimation program forestimating an own vehicle behavior that is a behavior of the vehiclebased on the image captured by the image sensor 3.

The scale factor corrector 12 learns a scale factor of the vehicle speedsensor 5 based on the signal Sa supplied from the vehicle speed sensor 5and the data Db representing the image captured by the image sensor 3.In the following description and FIG. 1 and the like, the scale factorcorrector is abbreviated as an SF corrector. The scale factor of thevehicle speed sensor 5 is a ratio of a detected value of the vehiclespeed sensor 5 to a vehicle speed to be measured by the vehicle speedsensor 5, that is, a ratio of an output change to an input change of thevehicle speed sensor 5, and is a coefficient for obtaining a true valueof the vehicle speed from the detected value of the vehicle speed sensor5. The SF corrector 12 detects the vehicle speed of the own vehiclebased on the signal Sa supplied from the vehicle speed sensor 5 and thescale factor corrected by learning, and outputs data Dc representing thedetected value to the ego-motion calculator 13.

The ego-motion calculator 13 estimates an own vehicle behavior that is abehavior of a vehicle based on an image captured by the image sensor 3,and corresponds to the behavior estimator. In addition, each processexecuted by the ego-motion calculator 13 corresponds to the behaviorestimation procedure. In this case, the ego-motion calculator 13 isconfigured to estimate the own vehicle behavior using the SFM method.The ego-motion calculator 13 includes an SFM recognizer 18 and a travellocus generator 19 configured by an SFM module.

Based on the data Db, the SFM recognizer 18 performs estimation ofego-motion which is a parameter representing the own vehicle behavior,that is, an attitude of the vehicle per se, and estimation of a distanceto a feature point to be described later. Note that the ego-motionincludes information indicating yaw, roll, pitch, and translation. Inthe above configuration, the image sensor 3 captures a surrounding imageof the vehicle while moving as the vehicle travels. The SFM recognizer18 extracts feature points that can be easily dealt with, such ascorners and edges, in an image of two viewpoints captured while theimage sensor 3 moves, that is, an image for two frames with differentimaging positions captured by one image sensor 3 at different timings.

The SFM recognizer 18 associates the feature points extracted in theimages for two frames, and calculates an optical flow of the featurepoints based on their positional relationship. For example, asillustrated in FIGS. 2 and 3, feature points P1 to P6 in an imagecaptured at a predetermined time t1, and feature points P1 to P6 in animage captured at a predetermined time t2, which is an image one frameafter that image, are associated with each other, and optical flows A1to A6 in which motions of the feature points P1 to P6 are represented byvectors are calculated. In FIG. 2, each feature point at time t1 isrepresented by a circle mark. In addition, in FIG. 3, each feature pointat time t2 is represented by a circle mark, and the position of eachfeature point at time t1 is represented by a white circle mark.

The SFM recognizer 18 estimates a three-dimensional position of eachfeature point and a posture of the image sensor 3, that is, ego-motion,using the plurality of optical flows calculated in this way. Inaddition, the SFM recognizer 18 can know a movement amount of the ownvehicle by such a method; however, there is a problem with the accuracyof the scale. Accordingly, the SFM recognizer 18 acquires a moving speedof the own vehicle based on the data Da representing the GPS informationand the data Dc representing the detected value of the vehicle speed,and improves the accuracy of the scale based on the moving speed.

The SFM recognizer 18 outputs data Dd representing the estimatedego-motion to the travel locus generator 19 and the landmark detector14. The travel locus generator 19 integrates the ego-motion estimated bythe SFM recognizer 18 every hour, and generates a travel locusrepresenting how the own vehicle has moved. The travel locus generator19 outputs data De representing the generated travel locus to the mapgenerator 15.

In the ego-motion calculator 13 configured as described above, ifaliasing occurs, there is a risk that the estimated own vehicle behaviorand the distance between each feature point may be different from theactuals. The aliasing referred to here is caused by an optical flow dueto an incorrect correspondence, that is, an optical flow calculated in astate where there is an error in the feature point correspondence. Inother words, in a location where there is an object having an appearancein which a same pattern is repeated along the travel direction of thevehicle, such as a soundproof wall or a tunnel, there may be a casewhere incorrect correspondence continues to occur in, a location wherethe same pattern is repeated, that is, a repeated pattern. Theabove-described location corresponds to a specific location where asituation around the vehicle is such that the estimation of the ownvehicle behavior based on the ego-motion calculator 13 becomes unstable.

The SFM optimizes an entire system, and thus if there is a small numberof occurrences of such incorrect correspondence, there is no significantimpact on the estimation of ego-motion; however, if the number ofoccurrences of erroneous correspondence increases, there is a risk thatthe estimation of ego-motion will fail. Hereinafter, a mechanism of suchaliasing will be described with reference to FIGS. 4 and 5. In thefollowing description, an interval between repeated patterns is alsoreferred to as a grid.

Here, as illustrated in FIGS. 4 and 5, the mechanism of aliasing will bedescribed by taking as an example a case where a vehicle is traveling ina section in which soundproof walls having the same shape arecontinuously provided. FIG. 4 illustrates an image captured at apredetermined time t11, and FIG. 5 illustrates an image captured at apredetermined time t12, which is an image one frame after the imageillustrated in FIG. 4. In this case, it is assumed that four featurepoints P11 to P15 are associated with each other for the images for twoframes in FIGS. 4 and 5. The interval between the repeated patterns inthis case is an interval between the wall edges of the soundproof walls.In FIGS. 4 and 5, some wall edges are denoted by a reference sign E.

Here, if the vehicle moves by two grids in one frame, a flow indicatedby arrows A11 to A15 is a correct optical flow. However, if a flowindicated by arrows A21 to A25 is dominant due to an incorrectcorrespondence, there is a risk that an erroneous estimation may be madethat the vehicle has advanced only by one grid in one frame, or that thefeature points P11 to P15 are far from the actual points. In addition,if a flow indicated by arrows A31 and A32 is dominant due to anincorrect correspondence, there is a risk that an erroneous estimationmay be made that the vehicle has sunk by one grid in one frame.Meanwhile, when an association that minimizes a vector is performed inthe SFM recognizer 18, an incorrect correspondence starts when “gridwidth≈vehicle movement amount”, and an influence increases as themovement amount increases.

Accordingly, in the present embodiment, the following refinement isadded in order to suppress the occurrence of such erroneous estimationdue to aliasing. That is, in this embodiment, the localizer 16 has afunction as the information acquirer that acquires beforehand specificlocation information that is information representing a specificlocation. As described above, the specific location is a location wherethere is an object having an appearance in which the same pattern isrepeated along the travel direction of the vehicle, such as a section inwhich soundproof walls having the same shape are continuously provided.

The specific location information includes information indicatingwhether the vehicle has passed through the specific location at thattime, the position of the abovementioned pattern (for example, theposition of the soundproof wall), and the abovementioned repetitioninterval of the pattern, that is, the interval of the repeated pattern(for example, the interval between the wall edges), and the like. Thelocalizer 16 outputs data Df representing the specific locationinformation to the ego-motion calculator 13. The SFM recognizer 18estimates the own vehicle behavior based on the data Df representing thespecific location information transmitted from the localizer 16 inaddition to the data Db representing the image captured by the imagesensor 3.

Specifically, the SFM recognizer 18 determines whether the vehicle haspassed through the specific location based on the specific locationinformation, that is, determines whether aliasing occurs. Whendetermining that aliasing occurs, the SFM recognizer 18 sets an imagingcycle of the image sensor 3, that is, an operation cycle of the SFM perse so as to fulfill a condition of the following formula (1). Note thata movement amount of the vehicle between one frame of the image capturedby the image sensor 3 is dx, and the repetition interval of the patternis w.

w/2>dx  (1)

That is, the SFM recognizer 18 sets the SFM cycle such that the movementamount dx of the vehicle is smaller than half the repetition interval ofthe pattern w, and, moreover, estimates the own vehicle behavior basedon the data Db representing the image captured by the image sensor 3. Inaddition, the SFM recognizer 18 only needs to set the cycle describedabove when the condition of the above formula (1) is not fulfilled, andwhen the condition of the above formula (1) is already fulfilled, thereis no need to perform the setting. By setting the SFM cycle in this way,the occurrence of erroneous estimation due to aliasing is suppressed.

In a general countermeasure against aliasing, due to that when an inputsignal having a frequency higher than half a sampling frequency issampled, aliasing occurs, the sampling frequency is set so as not tofulfill such a condition. The countermeasure against the occurrence oferroneous estimation due to aliasing in the present embodiment describedabove is also based on the same principle as that of such a generalcountermeasure against aliasing.

The landmark detector 14 includes a recognizer 20 and a target generator21. The recognizer 20 detects a position of a landmark on the imagecaptured by the image sensor 3 based on the data Db. Note that variousmethods can be employed as a method for detecting the position of thelandmark. The above landmark includes, for example, a sign, a signboard,a pole such as a utility pole or a streetlight, a white line, a trafficlight, the wall edge described above, and the like.

The recognizer 20 recognizes a travel path or lane of the own vehiclebased on the data Db, and acquires division line information that isinformation representing a road parameter and a division line. The roadparameter includes information representing a lane shape, such as a lanewidth and a curvature of the lane, that is, a road. In addition, theroad parameter includes information representing a traveling state ofthe vehicle relative to the lane shape, such as an offset representing adistance from a center position in a width direction of the lane to aposition of the vehicle and an yaw angle representing an angle betweenthe lane, that is, a tangential direction of the road and a traveldirection of the own vehicle.

In this case, travel path information such as the division lineinformation described above is also included in the landmark. Therecognizer 20 outputs data Dg representing such a landmark detectionresult to the target generator 21. The target generator 21 collates thedetected landmark with the SFM point in the detected landmark based onthe data Dg supplied from the recognizer 20 and the data Dd suppliedfrom the SFM recognizer 18, thereby obtaining physical positioninformation including a landmark distance and a lateral position. Thelandmark detector 14 outputs data Dh representing the road parameteracquired by the recognizer 20 to the vehicle controller 4. In addition,the landmark detector 14 outputs to the map generator 15 data Direpresenting information on a position of the landmark including travelpath information such as division line information generated by thetarget generator 21.

The map generator 15 generates map information based on the data Darepresenting GPS information, the data De supplied from the ego-motioncalculator 13, and the data Di supplied from the landmark detector 14.Specifically, the map generator 15 links the GPS information, thegenerated landmarks, and the travel locus, thereby generating mapinformation that is fragmentary map data. Hereinafter, the mapinformation generated by the map generator 15 is also referred to as aprobe map. Data Dj representing the probe map generated by the mapgenerator 15 is uploaded to the server 2 as probe data and is output tothe localizer 16.

The probe map has a limit in the accuracy of the SFM, and thus theaccuracy is not sufficiently high. Accordingly, the integrator 10 of theserver 2 superimposes and integrates a plurality of probe maps based onthe data Dj transmitted from a vehicle-mounted device of each vehicle,thereby improving the accuracy of the map. When the integration by theintegrator 10 is successful, the updater 11 of the server 2 updates theintegrated map. The server 2 distributes data Dk representing theintegrated map to the vehicle-mounted device of each vehicle. In thiscase, the server 2 identifies an approximate position of a distributiondestination vehicle based on the GPS information and the like, anddistributes an integrated map around the approximate position (forexample, a radius of several kilometers around the approximateposition). In addition, when a map exists on the vehicle-mounted deviceside, differential distribution of the map is also possible.

The localizer 16 performs localization for estimating the currentposition of the own vehicle. The localizer 16 downloads the data Dkrepresenting the integrated map from the server 2, and performslocalization on the integrated map based on the downloaded data Dk, thedata Dj representing the probe map, and the data Db representing theimage captured by the image sensor 3. In addition, the localizer 16 canalso perform localization without using the data Dj representing theprobe map.

The localizer 16 calculates the road parameter based on the mapinformation when the localization is successful. The localizer 16outputs data Dl representing a road parameter based on the mapinformation to the vehicle controller 4. The vehicle controller 4executes various processes for controlling the traveling of the ownvehicle based on the data Dh supplied from the landmark detector 14 andthe data Dl supplied from the localizer 16. That is, the vehiclecontroller 4 executes various processes for controlling the traveling ofthe own vehicle based on the road parameter.

In the above configuration, the map generator 15 has a function as apassage determiner that determines whether the vehicle has passedthrough the specific location and has a function as an informationtransmitter that transmits information related to the passed specificlocation (hereinafter also referred to as related information) to theserver 2 isolated from the vehicle when determining that the vehicle haspassed the above location. Each process executed by the map generator15, that will be described later, corresponds to a passage determinationprocedure and an information transmission procedure.

In this case, the map generator 15 determines whether the vehicle travellocus represented by the data De and the landmark informationrepresented by the data Di are physically impossible. Note that, as thetravel locus that is physically impossible, for example, a locus thatexhibits behavior exceeding a motion performance of the vehicle isassumed. In addition, as an example of landmark information that isphysically impossible, for example, there may be a sign size that is solarge or so small that it can not exist. If at least one of the travellocus and the landmark information is physically impossible, it ishighly likely that the vehicle is passing through a locationcorresponding to the specific location, that is, aliasing has occurred.

Accordingly, when determining that at least one of the travel locus andthe landmark information is physically impossible, the map generator 15transmits the information related to the specific location where thevehicle has passed at that time to the server 2. In this case, therelated information can be transmitted to the server 2 in a formincluded in the data Dj described above. The server 2 generates orupdates the specific location information based on the relatedinformation represented by the data Dj transmitted from a plurality ofvehicles.

The map generator 15 executes at all times such a process, that is, aprocess as the passage determiner and the information transmitter.Therefore, the map generator 15 executes the above-described eachprocess even when the vehicle passes through a specific locationcorresponding to the specific location information already acquired bythe localizer 16. In this way, the accuracy of the specific locationinformation generated in the server 2 can be raised.

In addition, on the vehicle-mounted device side, since it is difficultto determine the specific location with high accuracy, if the server 2generates the specific location information based on only the abovementioned related information, the accuracy may be lowered. Accordingly,the server 2 collects data such as a vehicle speed and a lane offsetdifference, and detects abnormal data that is inconsistent in the data.The server 2 checks the lane, the traveling lane, the own vehiclebehavior estimated at that time, and the like for the abnormal data, andcollates them with the integrated map created with normal data todetermine the occurrence of aliasing.

In this way, the server 2 detects a location where aliasing occurs, thatis, the specific location, and assigns specific location information,which is information about the specific location, to the data Dkrepresenting the integrated map. As described above, the server 2distributes data Dk to the vehicle-mounted device of each vehicle. Thelocalizer 16 having a function as the information acquirer acquires thespecific location information based on the data Dk distributed from theserver 2 and outputs the data Df representing the acquired specificlocation information to the ego-motion calculator 13.

Based on the estimated current position of the own vehicle, thepositional relationship between the current position and the recognizedlandmark, the vehicle speed of the own vehicle, the localizer 16determines whether the vehicle has passed through the specific locationat that time, that is, whether aliasing occurs. Therefore, thedetermination result may be included in the data Df transmitted to theego-motion calculator 13. In this case, the data Dk transmitted from theserver 2 may not include information indicating whether the vehicle haspassed through the specific location. In addition, the abovementionedeach process executed by the localizer 16 corresponds to the informationacquisition procedure.

In the present embodiment, as described above, a refinement forsuppressing the occurrence of erroneous estimation due to aliasing isadded; however there may be a case where the occurrence of erroneousestimation due to aliasing cannot be avoided. In such a case, theego-motion calculator 13 cannot estimate the own vehicle behavior. Then,there is a risk that a problem may arise in the process performed usingthe own vehicle behavior estimated by the ego-motion calculator 13.

Accordingly, the canceller 17 detects such an abnormality, that is, theoccurrence of an abnormality that makes it impossible for the ego-motioncalculator 13 to estimate the own vehicle behavior. For example, it isdetermined that the abnormality occurs when the vehicle is currentlypassing through a location corresponding to the specific location andinformation about the specific location (for example, the repetitioninterval of the pattern) is insufficient. In addition, depending on thevehicle speed and the repetition interval of the pattern, it may bedifficult to change the SFM cycle so as to fulfill the above formula(1); however, even in such a case, it can be determined that the aboveabnormality occurs. Moreover, the abovementioned each process executedby the canceller 17 and each process described later correspond to acancellation procedure.

When the occurrence of the abnormality is detected, the canceller 17cancels or changes the execution of a process related to the vehicle,that is performed using directly or indirectly the own vehicle behaviorestimated by the ego-motion calculator 13. As a result, a process ofeach functional block is stopped or changed as follows. That is, in thiscase, the SFM recognizer 18 is reset. Note that “reset” as used hereinmeans that the SFM recognizer 18 discards all the information on thefeature points acquired up to that point, and starts acquiringinformation on the feature points from the beginning.

In addition, in this case, since the correct data Dd is not transmittedfrom the SFM recognizer 18, the travel locus generator 19 usesinformation other than the data Dd and interpolates data based on thevehicle speed, yaw rate, GPS information, and the like on the vehicle,for example. By doing so, travel locus estimation with relatively lowaccuracy is performed. Moreover, in this case, since the distance cannotbe estimated correctly while the SFM recognizer 18 is not operating orthe SFM recognizer 18 is reset, the target generator 21 outputs alltargets in an invalid state so that they are not used in a post-process.

Alternatively, in this case, the target generator 21 executes a processby switching to a simple distance estimation logic that does not use theSFM or optical flow. As the simple distance estimation logic, variousmethods such as a method for estimating a distance based on the pinholeprinciple and a method for estimating a distance by assuming a landmarkcan be employed. However, the target generator 21 outputs in an invalidstate those that cannot be estimated by the simple distance estimationlogic (for example, a pole or the like).

In addition, in this case, the map generator 15 prevents the probe datafrom being uploaded by setting an output to an invalid value or uploadsonly an invalidity flag. Consequently, the integrator 10 of the server 2excludes the probe data to which the invalidity flag is assigned fromintegration targets. Further, in this case, since the travel locus andlandmark information are obtained by the above-described simple distanceestimation logic, the localizer 16 performs the localization processwithin a range that is possible by using the travel locus and landmarkinformation.

However, in this case, there is a high possibility that the localizationaccuracy is degraded. For this reason, the localizer 16 assigns to thedata Dl output to the vehicle controller 4 accuracy degradationinformation indicating that the localization accuracy is degraded.Consequently, the vehicle controller 4 executes various controls on thepremise of the accuracy degradation when the data Dl to which theaccuracy degradation information is assigned is transmitted. Forexample, in this case, the vehicle controller 4 executes a control witha control amount weakened, or stops the execution of a control thatrequires high accuracy.

As described above, the control device 8 of the present embodimentfunctions as an apparatus for estimating an own vehicle behavior basedon the image captured by the image sensor 3 that captures a periphery ofthe vehicle. In addition, the ego-motion calculator 13 included in thecontrol device 8 estimates the own vehicle behavior based on thespecific location information acquired beforehand by the localizer 16 inaddition to the image captured by the image sensor 3. According to sucha configuration, when the vehicle is traveling in the specific locationwhere the estimation of the own vehicle behavior based on the imagebecomes unstable, the ego-motion calculator 13 can supplement theestimation of the own vehicle behavior in the image based on thespecific location information. Therefore, according to theabovementioned configuration, an excellent effect that the own vehiclebehavior can be estimated accurately even in a location where theestimation of ego-motion in an image becomes unstable.

In this case, as the specific location, a location where there is anobject having an appearance in which the same pattern is repeated alongthe travel direction of the vehicle, such as a section in whichsoundproof walls having the same shape are continuously provided, isassumed. In addition, the ego-motion calculator 13 is configured toestimate the own vehicle behavior using the SFM method. For this reason,when the vehicle travels in the section as described above, aliasingoccurs, and there is a risk that an error may occur in the estimation ofthe own vehicle behavior by the SFM recognizer 18 of the ego-motioncalculator 13, that is, erroneous estimation due to aliasing may occur.

Accordingly, in the present embodiment, the specific locationinformation includes the position of the pattern and the repetitioninterval of the pattern. In addition, the SFM recognizer 18 sets the SFMcycle such that a movement amount of the vehicle per inter-frameinterval of the captured images captured by the image sensor 3 issmaller than half the repetition interval of the pattern, and, moreover,estimates the own vehicle behavior based on the image captured by theimage sensor 3. In this way, it is possible to suppress the occurrenceof erroneous estimation due to aliasing only by executing the processfor changing the SFM cycle without changing the process content forestimating the own vehicle behavior.

The control device 8 includes the canceller 17 that cancels or changesthe execution of a predetermined process related to the vehicle, that isperformed using directly or indirectly the own vehicle behaviorestimated by the ego-motion calculator 13 when the occurrence of anabnormality that makes it impossible for the ego-motion calculator 13 toestimate the own vehicle behavior is detected. According to such aconfiguration, even when the occurrence of erroneous estimation due toaliasing cannot be avoided, the above process is not executed as it isusing the erroneously estimated own vehicle behavior. Therefore,according to the above configuration, it is possible to prevent problemsin various processes performed using the own vehicle behavior, such as amalfunction of the vehicle controller 4, a decrease in accuracy of aprobe map generated by the map generator 15, and thus a decrease inaccuracy of the integrated map generated in the server 2, and a decreasein the localization accuracy by the localizer 16.

In the above configuration, the map generator 15 has a function fordetermining whether the vehicle has passed through the specific locationand transmitting the information related to the passed specific locationto the server 2 when determining that the vehicle has passed the abovelocation.

The server 2 generates the specific location information based on theinformation related to the passed specific location transmitted from aplurality of vehicles. According to such a configuration, the accuracyof the specific location information generated in the server 2 can beraised.

Second Embodiment

Hereinafter, a second embodiment will be described with reference toFIGS. 6 and 7.

In the present embodiment, the content of the process executed by theego-motion calculator 13, more specifically, the content of the processexecuted by the SFM recognizer 18 for suppressing the occurrence oferroneous estimation due to aliasing is different from the firstembodiment. When determining that aliasing occurs, the SFM recognizer 18of the present embodiment predicts a change in the position of thepattern on the image captured by the image sensor 3 based on theposition of the pattern and the repetition interval of the pattern, andestimates the own vehicle behavior using the prediction result.

Hereinafter, the specific content of a process for suppressing theoccurrence of erroneous estimation due to aliasing in the presentembodiment will be described with reference to FIGS. 6 and 7. Here, asillustrated in FIGS. 4 and 5, it is assumed that the vehicle istraveling in a section in which soundproof walls having the same shapeare continuously provided, and the process content for suppressing theoccurrence of erroneous estimation will be described using theassociation of a feature point P12 as an example.

In addition, FIG. 6 illustrates an image taken at a predetermined timet11 as is the case with FIG. 4. Moreover, in FIG. 6, a position of thefeature point P12 on the image one frame after is indicated by a whitecircle mark, and a dotted line indicating a center position of the ownvehicle is denoted by a reference sign C. Furthermore, in FIG. 7, aposition of the vehicle on the image captured at time t11 is M1, aposition of the vehicle on the image one frame after that is M2, a whiteline is WL, and a road edge is RE.

A repeated pattern width θ3 can be calculated from image coordinatesthat are coordinates on the image. In addition, a repetition patterninterval w is an interval between the wall edges and corresponds to therepetition interval of the pattern. That is, since being included in thespecific location information, the repetition pattern interval w can beacquired beforehand. Furthermore, a lateral position y from the centerof the vehicle to the road edge RE can be obtained from the mapinformation and localization, or from the detection result of the roadedge RE.

Assuming that the vehicle is traveling in parallel with the road edge REand a white line WL, a distance x along the travel direction of thevehicle up to the feature point P12 at time t11 can be calculated usingthe repeated pattern width θ3, the horizontal position y, and therepetition pattern interval w, which are known values as describedabove. In addition, an azimuth angle θ1 of the feature point P12 viewedfrom the vehicle at time t11 can be calculated from the imagecoordinates. Furthermore, the movement amount dx of the vehicle perinter-frame interval of the captured images can be calculated using thedetected value of the vehicle speed.

An azimuth angle θ2 of the feature point P12 after one frame can becalculated using the azimuth angle θ1, the movement amount dx, thedistance x, and the lateral position y, which are known values asdescribed above. In this way, if the position and interval of therepetitive pattern are clear, it is possible to predict the position ofthe feature point on the image after one frame, and the incorrectcorrespondence can be prevented by using the prediction result. Inaddition, the position and interval of the repetitive pattern describedabove are included in the specific location information acquiredbeforehand by the localizer 16.

The SFM recognizer 18 can determine whether each optical flow calculatedbased on the images for two frames is correct, based on the position ofthe feature point on the image after one frame predicted by theabove-described method. In this way, the SFM recognizer 18 can excludean optical flow determined to be incorrect as the result of thedetermination of being correct or incorrect and can estimate the ownvehicle behavior using only an optical flow determined to be correct.Alternatively, the SFM recognizer 18 can estimate the own vehiclebehavior after correcting the optical flow that is determined to beincorrect so to be a correct flow. Thus, the occurrence of erroneousestimation due to aliasing is suppressed.

As described above, also according to the present embodiment, theego-motion calculator 13 can supplement the estimation of an own vehiclebehavior in an image based on the specific location information.Therefore, also according to the present embodiment, as is the case withthe first embodiment, an excellent effect can be obtained that the ownvehicle behavior can be estimated accurately even in a location wherethe estimation of ego-motion in the image becomes unstable. Furthermore,according to the present embodiment, the following effects can also beobtained.

That is, in the present embodiment, since it is not necessary to changethe SFM cycle as in the first embodiment, the occurrence of erroneousestimation due to aliasing can be suppressed regardless of the vehiclespeed and the value of the repetition interval of the pattern.Therefore, according to the present embodiment, compared with the firstembodiment, it is possible to reduce the occurrence frequency of anabnormality that makes it impossible for the ego-motion calculator 13 toestimate the own vehicle behavior. In addition, since the technique ofthe first embodiment suppresses the occurrence of erroneous estimationdue to aliasing by changing the SFM cycle, it is impossible to deal witha case in which it is incorrectly estimated that the vehicle has sunk inone frame. However, according to the technique of the presentembodiment, it is possible to suppress the occurrence of erroneousestimation even in such a case.

Modifications

In addition, the present disclosure is not limited to each embodimentdescribed above and described in the drawings, and any modification,combination, or expansion can be made without departing from the scopeof the disclosure.

The numerical values and the like illustrated in the above embodimentsare examples and are not limited thereto.

In the map system 1, each functional block may be distributed. Forexample, a part of each functional block included in the control device8 on the vehicle side, that is, the vehicle-mounted device side may beprovided in the control device 9 on the server 2 side, and each controldevice may execute the estimation of the own vehicle behavior bytransmitting and receiving various data via communication.

In the first embodiment, while the SFM recognizer 18 changes the SFMcycle so as to fulfill the condition of the above-described formula (1),as an alternative to or in addition to this, the vehicle speed may bechanged so as to fulfill the condition of the formula (1). Even in thiscase, the action and effect similar to the case where the SFM cycle ischanged can be obtained.

Although the present disclosure has been described with reference to theembodiments, it is understood that the present disclosure is not limitedto the aforementioned embodiments and configurations. The presentdisclosure includes various variations and modifications within theequivalent range. In addition, various combinations and forms, as wellas other combinations and forms further including only one element, ormore or less than that, are within the scope and spirit of the presentdisclosure.

What is claimed is:
 1. An apparatus for estimating an own vehiclebehavior that is a behavior of a vehicle carrying the apparatus based onimages of surroundings of the vehicle captured by an imaging devicemounted to the vehicle, the apparatus comprising: an informationacquirer that acquires beforehand specific location information that isinformation representing a specific location in which a situation aroundthe vehicle is such that the estimation of the own vehicle behaviorbased on the images is unstable; and a behavior estimator that estimatesthe own vehicle behavior based on the images captured by the imagingdevice and the specific location information.
 2. The apparatus accordingto claim 1, wherein the behavior estimator estimates the own vehiclebehavior using a method of Structure From Motion, the specific locationis a location where there is an object having an appearance in which asame pattern is repeated along a travel direction of the vehicle, andthe specific location information includes a position of the pattern anda repetition interval of the pattern.
 3. The apparatus according toclaim 2, wherein the behavior estimator sets a cycle of the StructureFrom Motion such that a movement amount of the vehicle per inter-frameinterval of the captured images is smaller than half the repetitioninterval of the pattern, and estimates the own vehicle behavior.
 4. Theapparatus according to claim 2, wherein the behavior estimator predictsa change in the position of the pattern on the captured images based onthe position of the pattern and the repetition interval of the pattern,and estimates the own vehicle behavior using a prediction result.
 5. Theapparatus according to claim 1, further comprising a canceller thatcancels or changes execution of a predetermined process related to thevehicle, which is performed using the own vehicle behavior estimated bythe behavior estimator, directly or indirectly when an occurrence of anabnormality in which the own vehicle behavior cannot be estimated by thebehavior estimator is detected.
 6. The apparatus according to claim 1,further comprising: a passage determiner that determines whether thevehicle has passed the specific location; and an information transmitterthat transmits, to a server isolated from the vehicle, informationrelated to the specific location that has been passed when the passagedeterminer determines that the vehicle has passed the specific location,wherein the server generates or updates the specific locationinformation based on the information related to the specific locationthat has been passed transmitted from the information transmitter of arespective one of a plurality of vehicles, and the information acquireracquires the specific location information from the server.
 7. A methodfor estimating a behavior of a vehicle based on images of surroundingsof the vehicle captured by an imaging device mounted to the vehicle, themethod comprising: acquiring beforehand specific location informationthat is information representing a specific location in which asituation around the vehicle is such that the estimation of the ownvehicle behavior based on the images is unstable; and estimating the ownvehicle behavior that is the behavior of the vehicle based on the imagescaptured by the imaging device and the specific location information. 8.A non-transitory computer-readable medium storing a computer program,which when executed by a computer executes a method for estimating abehavior of a vehicle based on images of surroundings of the vehiclecaptured by an imaging device mounted to the vehicle, the methodcomprising: acquiring beforehand specific location information that isinformation representing a specific location in which a situation aroundthe vehicle is such that the estimation of the own vehicle behaviorbased on the images is unstable; and estimating the own vehicle behaviorthat is the behavior of the vehicle based on the images captured by theimaging device and the specific location information.
 9. An apparatusfor estimating an own vehicle behavior that is a behavior of a vehiclecarrying the apparatus based on images of surroundings of the vehiclecaptured by an imaging device mounted to the vehicle, the apparatuscomprising: a storage device storing a computer program; and a controldevice that reads and executes the computer program from the storagedevice, wherein the computer program, when executed, enables the controldevice to: acquire beforehand specific location information that isinformation representing a specific location in which a situation aroundthe vehicle is such that the estimation of the own vehicle behaviorbased on the images is unstable, and estimate the own vehicle behaviorbased on the images captured by the imaging device and the specificlocation information.